Skip to content

Point a race detector at the OpenMP code, and stop nested regions bypassing the plan - #14

Merged
MuteJester merged 2 commits into
mainfrom
parallelism/race-detector-and-nested-regions
Nov 16, 2026
Merged

MuteJester merged 2 commits into
mainfrom
parallelism/race-detector-and-nested-regions

Conversation

@MuteJester

Copy link
Copy Markdown
Owner

Two pieces of work on the parallelism core. They touch disjoint files, so each
commit carries its own evidence: the detector commit moves no recorded number
and no inventory line, and the nesting commit carries all three inventory
lines and the whole RERECORDED.md section.

Run a race detector over the OpenMP code (7fb368f)

93 parallel regions and no race detector had ever been pointed at one. The two
checks that exist are static -- thread_count_clauses proves every region
NAMES a count, the GIL scan proves every GIL-releasing region RESOLVES a plan.
Neither says anything about what two threads do to the same address.

make tsan builds clang + libomp into build-tsan/; tools/qa/run c --tsan
runs it. 33 suites, 368 tests, 0 races. Three blockers had to fall first,
and each one would have produced a confident wrong answer on its own:

  • Archer is not optional. libomp carries no sanitiser annotations, so
    without an OMPT tool TSan sees no happens-before edge between any two OpenMP
    threads: every implicit barrier is invisible and every region whose result
    somebody later reads is a race. Same binary, one suite at 8 threads: 783
    reports with Archer off, 0 with it on.
    Under GCC and libgomp, where there
    is no Archer at all, 1186. So the mode refuses to run when it cannot find
    libarcher.so, and the last test runs a suite with the tool unset and
    requires the reports to come back -- a clean run is only evidence if a dirty
    one is reachable.
  • target_clones and TSan cannot coexist. The attribute emits an IFUNC
    resolver, the loader calls it during startup relocation before the sanitiser
    runtime initialises, and clang instruments the resolver, so
    __tsan_func_entry runs against uninitialised state and the process dies
    inside ld.so before main, printing nothing. Nine of 35 suites: exit 139,
    empty log. The three guards now carry && !BEAM_TSAN.
  • TSan refuses to start under an address-space ceiling -- it maps shadow
    memory across the address space and setrlimit() fails with 22. So
    verification/bounded cannot wrap it; the mode swaps it for a bare timeout
    plus max_allocation_size_mb and allocator_may_return_null=1.

Four suites stay uninstrumented, all four about the allocator, which is the one
thing TSan replaces wholesale. The test derives that set from the sources
rather than listing it, so a fifth cannot be quietly dropped from the run.

Nothing in CI runs this: it needs a clang toolchain, and today's answer is "no
races on this machine today", not a standing one. verification/README.md says
so, and says what the binaries mean if you run them any other way.

Ask for one thread when the region is a nested one (be77719)

beam_threads() consulted the thread-local plan and, failing that, the
configured cap. The failure mode is not that the plan gets multiplied -- it is
that the plan is not consulted at all. The inner pragmas are encountered by
the OUTER team's workers, and a worker holds no plan (thread-local
deliberately: a process-global one would be visible to workers, but two BEAM
calls on two Python threads would then overwrite each other's plan mid-call).
So a worker fell through to threads_cap(), which with no configured cap is
omp_get_max_threads(). An 8-thread plan opened eight teams of sixteen.

Measured on one entropy() call, plan pinned at 8, OMP_MAX_ACTIVE_LEVELS=2:
117 threads in the process against 9 with nesting off, 0.24 s against under a
millisecond.
After the fix, 9 either way, entropy 17.6075259553 in all four
runs. Phase 0 measured that class of oversubscription at 301x-2275x worse than
serial on a loaded cluster node, which is what makes it worth a behaviour
change rather than a note in the docs.

  • omp_set_max_active_levels is not the answer: process-global state that two
    BEAM calls on two threads would fight over, the same reason the plan is
    thread-local. A thread's own level is the one thing always true locally.
  • ACTIVE level, not level. A team of one is an inactive region; it
    multiplies nothing, and omp_get_level() would serialise a single-length
    sweep for no gain. A test pins that choice against a later simplification.
  • Safe for the scratch sizing, in the direction that matters: a resolved count
    both sizes buffers and sets num_threads, so the dangerous disagreement is a
    buffer SMALLER than its team. This can only shrink the answer as nesting
    deepens.

One existing assertion changed, in
test_a_plan_does_not_reach_another_thread -- the part to scrutinise. The test
keeps its name and its property, and the cap is still configured and still
required to differ from the plan, but the fallback it recorded was the defect
itself. RERECORDED.md quotes the diff and says why.

Verification

detector nesting
races, this tree 0 over 33 suites / 368 tests same run
detector proven to bite 783 reports with Archer off --
C suite 452 passing 454 passing
Python -- 1243 passed, 2 skipped
tooling 13 new, 11 written red first 130 passed
gate exit 0, no line moved exit 0, every number byte-identical
ASan on the resized path -- 8 suites, 116 tests, 0 errors
mutation Archer-off run reports fails without the fix: 3.74 s, 109 threads over plan

thread counts (93 parallel regions, every one names a permitted thread count)
is OK in the same gate run: the clauses are untouched, only what one of them
evaluates to on a thread already inside a team has changed.

The shipped build is unaffected by the detector commit -- BEAM_TSAN
preprocesses to 0 without the flag, and a forced rebuild gives a
byte-identical libbeam.a and .so. The 11 toolchain-free tooling tests pass
with build-tsan/ absent and the other 2 skip, which is the state every runner
is in.

Not addressed here

  • Nothing observes an actual team width at runtime; both existing checks are
    static.
  • TSan is not in CI, so the zero is not standing.
  • beam_budget_concurrency's per_item == 0 early return skips the
    beam_threads() cap below it. Latent: all 17 call sites pass a computed size
    that is 0 only for a 0-state model.

BEAM has 93 parallel regions and no race detector had ever been pointed at
one of them. The two checks that exist are static: thread_count_clauses
proves every region NAMES a thread count, the GIL scan proves every
GIL-releasing region RESOLVES a plan. Neither says anything about what two
threads do to the same address.

`make tsan` builds clang + libomp into build-tsan/, `tools/qa/run c --tsan`
runs it. 33 suites, 366 tests, 0 races, 782 s. Three things had to be fixed
before that number existed, and each one would have produced a confident
wrong answer on its own.

Archer is not optional. libomp carries no sanitiser annotations, so without
an OMPT tool TSan sees no happens-before edge between any two OpenMP threads:
every implicit barrier is invisible and every region whose result somebody
later reads is a race. Same binary, one suite at 8 threads: 783 reports with
Archer off, 0 with it on. Under GCC and libgomp, where there is no Archer at
all, 1186. So `--tsan` refuses to run when it cannot find libarcher.so rather
than reporting a thousand races, and the last test in
verification/tests/test_tsan.py runs a suite with the tool unset and requires
the reports to come back -- a clean run is only evidence if a dirty one is
reachable.

target_clones and TSan cannot coexist. The attribute emits an IFUNC resolver,
the dynamic loader calls it during startup relocation before the sanitiser
runtime initialises, and clang instruments the resolver, so __tsan_func_entry
runs against uninitialised state and the process dies inside ld.so before
main, printing nothing. Nine of 35 suites: exit 139, empty log. The three
guards now carry `&& !BEAM_TSAN`, which single-versions those functions under
instrumentation and leaves the shipped build untouched -- BEAM_TSAN
preprocesses to 0 without the flag, and the forced rebuild gives a
byte-identical libbeam.a and .so.

TSan refuses to start under an address-space ceiling: it maps shadow memory
across the address space and setrlimit() fails with 22. So verification/bounded
cannot wrap it, and `--tsan` swaps it for a bare timeout plus
max_allocation_size_mb and allocator_may_return_null=1, which do the same job
from inside the process.

Four suites stay uninstrumented, all four about the allocator, which is the
one thing TSan replaces wholesale -- they fail on their own premise, reporting
that the policy did not reach the binary and that resident memory is many
times the model's size. The test derives that set from the sources
(statm|VmRSS|mallopt|M_MMAP_THRESHOLD) instead of listing it, so a fifth such
suite cannot be quietly dropped from the run.

-mavx is on the sanitiser build because clang passes a 32-byte vector by
value in vtrig.c, and with the clones off that is a -Wpsabi warning rather
than an error. The flag makes the instrumented build agree with the shipped
one about that ABI and keeps the log clean.

Nothing in CI runs this; it needs a clang toolchain and today's answer is "no
races on this machine today", not a standing one. verification/README.md says
so, and says what the binaries mean if you run them any other way.

Verified: 13 tooling tests, 11 of them written red first; C 452 tests
passing, gate exit 0, every recorded number byte-identical, no inventory line
moved by this commit.
beam_threads() consulted the thread-local plan and, failing that, the
configured cap. Neither is what a nested region should get, and the failure
mode is not that the plan gets multiplied -- it is that the plan is not
consulted at all.

The inner pragmas are encountered by the OUTER team's workers, and a worker
holds no plan. The plan is thread-local deliberately: a process-global one
would be visible to workers, but two BEAM calls on two Python threads would
then overwrite each other's plan mid-call, which is the buffer-versus-team
mismatch the mechanism exists to prevent. So a worker fell through to
threads_cap(), which with no configured cap is omp_get_max_threads(). An
8-thread plan opened eight teams of SIXTEEN.

Measured on one entropy() call with the plan pinned at 8 and nesting enabled
from the environment with OMP_MAX_ACTIVE_LEVELS=2: 117 threads in the process
against 9 with nesting off, 0.24 s against under a millisecond. After the fix,
9 either way. Phase 0 measured that class of oversubscription at 301x to
2275x worse than serial on a loaded cluster node, which is what makes it worth
a behaviour change rather than a line in the documentation.

omp_set_max_active_levels is not the answer. It is process-global state that
two BEAM calls on two threads would fight over -- the same reason the plan is
thread-local. A thread's own level is the one thing that is always true
locally.

ACTIVE level, not level. A team of one is an inactive region; it multiplies
nothing, and reading omp_get_level() there would serialise the work inside a
single-length sweep for no gain.
test_an_inactive_region_does_not_lose_the_plan pins that choice against a
later simplification.

Safe for the scratch sizing, and in the direction that matters. A resolved
count both sizes buffers and sets num_threads, so the dangerous disagreement
is a buffer SMALLER than its team. This can only shrink the answer as the
nesting deepens, and a region entered at a given level is sized by a read at
that level or deeper, never by a shallower one. math/spectrum.c's per-thread
scratch, allocated inside a nested call, was being sized for
omp_get_max_threads() and used by one thread; eight suites on that path
under AddressSanitizer, 116 tests, report nothing.

One existing assertion changed, in test_a_plan_does_not_reach_another_thread,
and RERECORDED.md says why at length: the test keeps its name and its
property, the cap is still configured and still required to differ from the
plan, but the fallback it recorded was the defect itself.

Verified: C 454 tests passing, Python 1243 passed and 2 skipped, tooling 130
passed, gate exit 0 with every recorded number byte-identical -- including
`thread counts`, the check this might have been expected to disturb, since
the clauses are untouched and only what one of them evaluates to on a thread
already inside a team has changed. The Python test is mutated against the
real build: with the line commented out and the extension rebuilt it fails at
3.74 s with the process 109 threads above its plan; restored, 0.35 s. Two
inventory files moved by three lines in total, named in RERECORDED.md.
@MuteJester
MuteJester merged commit 17cda19 into main Nov 16, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant